Tsinghua and CUHK researchers open-source AKV, cutting LLM agent costs by 27.7%

Researchers from Tsinghua University and the Chinese University of Hong Kong have open-sourced AKV, an inference serving system built for key-value cache engineering in large language model agents. Instead of summarizing older interaction history, AKV lets agents discard and relocate KV cache entries. In tests covering three models and four agent benchmarks, the system reduced average cost by 27.7% and improved average task scores by 1.7 points. The code is released under the Apache-2.0 license, built on top of SGLang, and runs on NVIDIA GPUs. The research paper used eight A800-80GB GPUs. According to the reported results, AKV delivered its best performance with the GPT-OSS-120B model on the SWE-bench Verified benchmark, where per-task cost fell from $0.20 to $0.097. The development was reported by MarkTechPost.